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Abstract. - We propose a method for computing the Kolmogorov- Sinai (KS) entropy of 
chaotic systems. In this method, the KS entropy is expressed as a statistical average over 
the canonical ensemble for a Hamiltonian with many ground states. This Hamiltonian is 
constructed directly from an evolution equation that exhibits chaotic dynamics. As an example, 
we compute the KS entropy for a chaotic repeller by evaluating the thermodynamic entropy of 
a system with many ground states. 



Chaotic systems are characterized by properties of 0(e N ) distinguishable trajectories over 
a time interval TV in a phase space in which the limit of observational accuracy is given by 
some finite value e [1-3]. In particular, if these trajectories are coded as sequences of symbols, 
the optimal compression ratio of such sequences is an important quantity from the point of 
view of information theory [4,5]. This quantity corresponds to the Kolmogorov-Sinai (KS) 
entropy, h ks . 

Although the KS entropy provides a simple characterization of chaotic systems, its direct 
numerical evaluation is not simple. One reason for this is that a precise mathematical definition 
of distinguishable trajectories is difficult to investigate numerically [1]. However, this difficulty 
can be overcome when a set of periodic orbits alone can provide information that is sufficient 
to obtain a useful characterization of the set of all distinguishable trajectories [6-8]. However, 
even in such cases, the difficulty involved in numerical computations remains, because the 
number of periodic orbits of period N increases exponentially as a function of N. 

Here, let us recall that in the statistical mechanics of glassy systems [9] and decision 
problems [10], the number of explored states increases exponentially as a function of the 
system size. Recently, efficient methods for computing thermodynamic quantities of glassy 
systems have been developed [11,12] and applied to several problems [13-15]. Inspired by 
this success, in this paper, we propose a method for computing the KS entropy employing a 
quantity that can be calculated in a system with many ground states. 
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Specifically, we consider a map r n+ i = F(T n ), with {T n } £ C R d , and assume that its 
trajectories can be coded with sequences of D symbols. The frequency distribution of an TV 
symbol sequence [s k ] = (s k ,s k , ■ • ■ , Sjy), with the natural measure for the initial conditions, 
is denoted by [i([s k ]), where € {1, ■••,£>}, and k is the index identifying the sequence. 
Letting Af(TV) be the total number of possible symbol sequences, the KS entropy, hus, is 
defined in terms of the Shannon entropy rate [4] as 

M(N) 

hus^-Km- J2 M([^])logM([s fc ]). (1) 

k=l 

Note that h^/ log(-D) represents the optimal compression ratio of symbol sequences [5]. 

In order to study simple cases, we further assume that for each [s k ] there exists an TV- 
periodic orbit T k , Iji, • • • , rjy, such that the symbol corresponding to r* is s k for all i < N. 
Then there is a one-to-one correspondence between the set of all symbol sequences {[s fc ]} and 
the set of all TV-periodic orbits {[L fe ]}- Furthermore, when TV is sufficiently large, /^([s fc ]) is 
accurately approximated by the probability 

„. = e -NKir k ]) (o) 

where A([L fc ]) is the volume expansion ratio in the unstable directions around the periodic 
orbit [r fc ]. The normalization constant is determined by the condition X)fc=i Pk = 1- I 11 
terms of pk , the KS entropy is expressed as 

M(N) 

hhs = - hm — V p k \ogpk- (3) 

N—>oo 1\ * — J 
k=l 

Note that the relation limjv^oo log^d(TV)/TV = holds for many physically interesting sys- 
tems. In this case, the expression (0 leads to the useful relation (Pesin's formula) that hk s 
is equal to the sum of the positive Lyapunov exponents [1,3]. However, there are systems 
to which this formula cannot be applied. A typical example is a chaotic repeller. Because 
the KS entropy of a repeller is related to transport properties of nonequilibrium phenomena 
described by deterministic models [16], it is important to study a computational method for 
/iks on the basis of |J3J that does not employ Pesin's formula. 

Basic formula:. - Here and hereafter, we use [r] = (ri, • ■ • , Tjv) to represent an array of 
TV phase space points, not a trajectory given by the map r n+ i = F(T n ). We also introduce 
ro as To = Tjv for later convenience. Let A m ([r]), with 1 < m < d, be the eigenvalues of 
the matrix F'(Tn)F'(Tn^i) ■ ■ ■ F'(Ti), where F'(T n ) is the d x d Jacobi matrix determined 
from the map L„+i = F(T n ). The volume expansion ratio A([r fc ]) in the unstable directions 
around the periodic orbit [T k ] is defined as 

A([r fc ])^l J2 iog|A m ([r fc ])|. (4) 

m;|A m ([r fc ])|>l 

Here, we assume that all periodic orbits are hyperbolic and hence satisfy |A m | ^ 1 for 1 < m < 
d. The volume expansion ratio A([r]) for an arbitrary array [r] can be defined by replacing 
T k in JU with L. 
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For an arbitrary function A([r]), we define the quantity 

M(N) 

1(A) ee A([T k ])e- NX ^\ 



Substituting J2J into J21 and using we rewrite /ik s as 

h ks = lim 



JW 'i.o 6 /(l) 



7(1) N 



(5) 



(6) 



We now seek to express 1(A) in a form that can be computed easily. We first rewrite (JSJ) as 



M(N) 

HA) = £ 

fe=i 



[] y «(r„ - it) 



-NX([r]) 



(7) 



Then, defining </>„([r„]) = T n — F(r„_i), n = 1, 2, • • • , N, we obtain the formula 



JV 

n 



where 



J([r]) = dct 



\J([F})\ 
d((j)i,- ■ ■ ,0n) 



(8) 



(9) 



0(Ti > ...,r JV ) 

Here, we note that J([r' c ]) 7^ for all hyperbolic periodic orbits, because in this case it can 
be proved that 

d 

J(\fl)= II (i-A m ([r])). (10) 

m— 1 

Also, from the expression . we can derive 



|J([r fe ])| =e m (f]>+«M, 



Then, expressing the delta function as 

S(x) = lim \ -e~ Px \ 
and ignoring the sub-exponential contribution, we can rewrite 1(A) as 



Nd/2 



i(A) = lim I - ) / dv 1 ■ ■ ■ dr A r e - /3ff([r]) ^([r]), 

/3^oo \ TT J J 



with 



JV 



H({T}) = Y,\rn-F(r n _ 1 )f 



n=l 



The expression (|13f) leads us to introduce a canonical ensemble in the form 



Pc([T}) = 



Z((3) 



(11) 



(12) 



(13) 



(14) 



(15) 
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with the partition function 

Z{(3) = J dT 1 ---dT N e-P H ^\ (16) 



Then, (|13[) becomes 

Nd/2 



1(A) = lim (tL) (A) p Z((3), (17) 

where ( } „ represents the average taken with respect to (I15|l . Substituting this result into 0, 
we obtain the formula 



/iks = lim lim 

AT — >oo f3 — >oo 



(A>, + ^log(^)+llogZ(/3) 



(18) 



Recall that the Hamiltonian (|14fl is non-negative and is equal to zero when [r] coincides with 
a periodic orbit \T ]■ Also, the number of periodic orbits M(N) is of order e N for a chaotic 
map F. These facts imply that the KS entropy, hk s , can be obtained using the formalism of 
statistical mechanics with the Hamiltonian 114|) . which possesses 0(e N ) ground states. 

Computational method:. - Let us define the entropy function S(E) through the relation 
e s(E) dE = J dTt ■ ■ ■ dT NE < Hm) < E+dE . (19) 
We then evaluate the partition function Z(j3) as 

Z((3) = e S(E,(f3))-/3E4(3)+0(log(N))^ (20) 

where E*([3) is defined as 

Then, from a standard argument employed in statistical mechanics, the relation E*((3) = (H)a 
is obtained, and the quantity S(f3) = S(E*(f3)) is calculated from the identity 

-4f>=C<«. (22) 

where C((3) represents the heat capacity defined by 

C{(3)^(i 2 {{5Ef) p . (23) 

Here, because = lim^oo C((3) takes a nonzero value, S((3) diverges as ~ log/3 in the 
limit (3 — ► oo. Therefore, in order to avoid a singularity, we define So(P) according to 

S((3) = -C oo 6((3-l)\og(3 + S ((3), (24) 

where 9( ) is the Heaviside function. Then, from 1|22[) and l|24|l . we can express So((3) in terms 
of C(J3), using the condition 

e S (0)+o(N) = | fi |iV_ (25) 
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Combining this result with 1|2U[) and l|24|). we rewrite the second and third terms of 118|) as 



da- 



ft 



(26) 



where we have used the relation lim^oo /3E*(f3) — Cqc — Nd/2. Using this quantity, the KS 
entropy given by (|18|l can be expressed as 

h ks = Jim [ lim (A)^ - f(N)}. (27) 

Therefore, the KS entropy is obtained from (A)^ and C(ft), which can be computed using the 
canonical ensemble (|15f) with the Hamiltonian Q14JI. 

As one method to obtain the canonical ensemble, we consider the Langevin equation 

f.w =-™+«~ m 

where represents Gaussian white noise satisfying 

(£n{t)ZmV)) = 2p~ X S{t - t')S nm . (29) 

The steady distribution of arrays [r] in this system is given by the canonical ensemble l|15l) 
with the Hamiltonian l|14l) . However, because there are 0(e N ) ground states of H, it is difficult 
to obtain accurately the canonical ensemble l|15|l by direct numerical integration of l)28|) when 
ft is large. For this reason, we employ an exchange Monte Carlo method [12] in order to 
improve the accuracy of the numerical computations. Below we describe this method. 

We first prepare L copies of the Langevin system. Each system is subject to noise with a 
time-dependent inverse temperature ft^\t) that takes values in the set {^o^, ' ■ • , fto^}- We 
have the initial values /?W(0) = ft^ \ and the time evolution of each ft^(t) is described by 
the following rule: At time kr with k = 1,2, • • •, two inverse temperatures ft^' and ft^ ', 
given by 0& = ft^ ] and ft^"> = ft ( Q t+1) for some i satisfying i = (1 - (-l) fc )/2 + 2i' - 1, with 
i' = 1,2, ■ • •, are exchanged with probability 

(30) 



1 + e (/3^ +1) -/3<")(ff([r]('))-H([r]('+i))) 

where [r] ^ is the configuration of the system subject to the noise with the inverse temperature 
fty \ It can be shown that the steady distribution of ([T^ 1 ), • • • , [T]^)) satisfies 

P c ([r]W, • • ■ , [rp)) = le-Et^dr]^ (31) 

z 

where Z is a normalization constant. It has been demonstrated that the exchange Monte Carlo 
method is useful for calculating thermodynamic quantities in systems with many metastable 
states [9], and therefore we expect that this method will be effective in application to the 
present problem. 

Example:. - As a simple demonstration of the computational method described above, 
we study a map F given by 

4 

z n +i = z n + - cos(x n + z„), (32) 
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Fig. 1 - Heat capacity as a function of log 0. The circles, squares, triangles, stars and pluses represent 
the results for the cases N = 4, 8, 12, 16 and 20. 



Fig. 2 - (A)^ as a function of log/3. Here, N = 20. 



which is defined in the region SI = {(x, z)|0 < a; < 27r,0 < x + z < 2ir} C R 2 . This 
map is obtained as an approximation of a two-dimensional map that describes a bouncing 
ball on a vibrating table [2]. It also becomes the so-called standard map under a variable 
transformation. We wish to compute the KS entropy of the repeller of SI, which is defined as 
the set {r| J F"(r) € SI for all n € Z}. 

Specifically, we investigate the case g = 0.3. In the numerical simulations, we set r = 
1CT 3 and f3 { l) = l.^ -8 ), where 1 < i < L = 30, with a time step At = 10~ 5 for the 
integration of The heat capacity, C(/3)/N, is plotted for N = 4, 8, 12, 16 and 20 in Fig. 

n From the value of the heat capacity thus obtained numerically, we derive the estimation 
liniTv^oo f(N) = 1.8(3). Furthermore, (X)g as a function of (3 is displayed in Fig. |21 Using 
this result, we derive (A)^ = 2.5(0). Thus, from l|26|) . we obtain h^ s = 0.6(7). 

Let us check the validity of our computational result. First, because the escape rate formula 
holds for hyperbolic maps [17], the value liiriAr^oo f(N) should be equal to the escape rate, 7. 
Directly computing the escape rate, we obtain 7 = 1.8(3). Thus, our computational result is 
consistent with the escape rate formula. Second, in this model, we can directly evaluate the 
right-hand side of Q by using the binary symbols and 1 for x < ir and x > ir, respectively. 
Collecting many segments of trajectories on the repeller over the time intervals —N < n < N, 
with for N = 1, 2 and 3, we obtain h^ a = 0.6(8). This value is also consistent with that found 
using our method. 

Discussion:. - Let us finally comment on topics related to the basic formula Ijl8jl, First, 
the statistical mechanical formulation used in the present paper is not related to the so-called 
thermodynamic formalism of chaos [4], in which, instead of ©, the probability 

Pk = = e-^W*!) (33) 

is assumed for each periodic orbit. In a manner similar to that used here, Zd(f3,N) can be 
obtained with a statistical mechanical formulation, but the Hamiltonian for cases in which 
$ 1 includes the Jacobian J([F]), and this makes it difficult to carry out the computation. 

Second, it is well known that the chaotic trajectories of the standard map correspond to 
the stationary configurations of the Frenkel-Kontorowa model [18]. However, such a corre- 
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spondence can be found only for a certain class of maps. Also, even for the standard map, 
it seems difficult to develop a method for computing the KS entropy by using this correspon- 
dence, because periodic orbits correspond to local minima and maxima, not ground states of 
the Frenkel-Kontorowa Hamiltonian. Nevertheless, it is worth noting that this correspondence 
has led to a mathematical study to evaluate the lower bound of the topological entropy [19]. 

Finally, let us recall that the expression 1)18(1 is valid in the case that none of the eigenvalues 
A m (1 < m < d) of the periodic orbits is equal to 1. However, even if there is a non- hyperbolic 
periodic orbit whose eigenvalue A m is unity, the right-hand side of the expression (|18|) can 
be computed. It might be an interesting problem to investigate the expression (|18|) for non- 
hyperbolic systems. 

In conclusion, we derived the formulae l|18|) and (|27|l for the KS entropy. We then applied 
these formulae to compute the KS entropy for the case of the repeller of a simple map by 
using the corresponding Langevin equation together with the exchange Monte Carlo method. 
We hope that these formulae can be similarly applied to many problems. 
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